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ABSTRACT 

One  of  the  biggest  challenges  in  developing  TRA  is  the  determination  of  the  critical  technology  elements  (CTEs). 
This  paper  outlines  a  systematic  process  used  to  identify  all  potential  technologies  developed  by  the  contractors 
during  the  Technology  Development  (TD)  Phase  and  applies  criteria  for  selection  of  CTEs.  To  reduce  the  subjectivity 
in  the  assessment ,  the  relevant  technical  requirements  for  each  CTE  that  are  important  to  the  customers  will  be 
established.  These  requirements  must  be  met  to  demonstrate  the  level  of  maturity  required  before  entering  the 
Engineering  &  Manufacturing  Development  (EMD)  Phase.  The  maturity  of  a  CTE  cannot  be  evaluated  in  isolation. 
The  paper  also  includes  other  system  requirements  that  the  CTEs  must  satisfy  before  the  overall  system  can  be 
evaluated.  A  major  defense  ACAT  vehicle  acquisition  program,  Joint  Light  Tactical  Vehicle  (JLTV),  was  used  to 
demonstrate  the  TRA  process  in  preparation  for  the  MS-B  Review  with  the  Assistant  Secretary  of  the  Army  for 
Acquisition,  Logistics,  and  Technology  (ASA(ALT))  and  the  Assistant  Secretary  of  Defense  for  Research  &  Engineering 
(ASD(R&E))  within  the  Office  of  the  Secretary  of  Defense  (OSD). 


Background 

Because  of  its  limited  budget,  the  US  Government 
has  to  select  technologies  that  are  mature  enough  to 
develop  into  products  among  the  many  immature 
technologies.  All  technology  developments  have 
cost  and  time  constraints.  Since  the  beginning  of  the 
war  in  Iraq  and  Afghanistan,  the  US  Army  (USA)  and 
US  Marine  Corps  (USMC)  wanted  to  upgrade  the 
protection  and  other  needed  capabilities  of  its  light 
tactical  High  Multipurpose  Mobility  Wheeled  Vehicle 
(HMMWV).  In  2007,  the  USA/USMC  selected  3 
vendors  to  begin  developing  a  Joint  Light  Tactical 
Vehicles  Family  of  Vehicles  (JLTV  FoV)  to  replace  its 
aging  HMMWV  fleet.  The  all  new  JLTV  has  to 
balance  conflicting  requirements  between 
protection,  performance  and  payload.  Its  main 
objectives  are  to  protect  soldiers  from  mines  and 
improvised  explosive  devices,  move  fast  and  reliable 
enough  to  perform  tactical  missions  in  rough  terrain, 
and  be  light  enough  to  be  transported  and  powerful 
enough  to  carry  heavy  loads.  In  addition,  it  must  be 
produced  affordably  at  low  risk  constrained  by  its 
program  timing.  Achieving  these  goals  require 
technologies  to  be  fairly  mature  and  affordable. 
Because  JLTV  is  an  Acquisition  Category  ID  program, 
the  Program  Management  (PM)  has  to  follow  the 


TRA  Guidance  released  by  OSD  in  determining  if  the 
technologies  are  critical  and  mature  enough  to  be 
implemented  into  the  vehicle.  For  the  last  18 
months,  the  US  Army  Tank  Automotive  Research  and 
Development  Engineering  Center  (TARDEC) 
supported  JLTV  in  selecting  the  CTEs  and  assessing 
their  technical  risks.  Prior  to  MS-B  Review,  based  on 
DoD  Directive  5000.2,  all  system  prototypes  shall 
have  been  demonstrated  in  a  relevant  environment 
satisfactorily  to  be  considered  mature  enough  to  use 
for  product  development.  The  purpose  of  this  paper 
is  to  summarize  the  systematic  processes  used  in 
selecting  and  evaluating  the  maturity  of  CTEs  and 
the  lessons  learned  during  the  process. 

Process  Timing 

The  original  estimate  was  to  complete  the  whole 
assessment  process  in  9  months.  The  actual  time  it 
took  was  16  months.  Table  1  shows  the  comparison 
between  the  estimated  and  the  actual  time  required 
to  complete  the  major  tasks.  For  MS-A  Review 
however,  much  less  information  was  available  for 
review  and  therefore  the  time  required  to  review  it 
should  also  be  proportionately  less. 
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Table  1 TRA  Time  Estimate  versus  Actual 


Technology  Maturity 
Assessment  Tasks 

Estimate 

(days) 

Actual 

(days) 

Project  planning 

15 

15 

Scope  of  work 

3 

3 

Collect  project  data 

20 

50 

Extract  technology 

15 

25 

Form  SME  Team 

5 

10 

Link  technology  &  KPP 

2 

3 

Establish  CTE  criteria 

5 

3 

Initial  CTE  screening 

5 

50 

PM/ASA(ALT)  Review 

10 

OSD  CTE  deliberation 

20  parallel 

CTE  &  TMA  process  brief 

10 

Collect  TRL6  requirements 

20 

SME  Tailor  TRL  checklist 

8 

10 

Combine  CTE  TRL  checklist  & 
TRL6  technical  requirements 

4 

5 

SME  TRL6  metrics  review 

5 

PM  TRL6  metrics  review 

10 

Test  data  collection 

95  parallel 

Organize  CTE  data 

15 

Assess  CTE  technical  risks 

15 

20 

Collect  SME  risk  assessments 

4 

PM  review  of  assessments 

5 

Finalize  CTE  assessments 

10 

Fill  in  TRL  checklist 

95  parallel 

Calculate  TRL/MRL/PRL 

20 

5 

Summarize  all  assessments 

10 

10 

ASA(ALT)  TRL  &  BOE  review 

10 

Write  draft  TMA  report 

50 

30 

Internal  reviews 

5 

5 

Finalize  TMA  Report 

10 

10 

Total 

192d=38wk 

353d=71wk 

=9m 

=16m 

Lesson  ft 1 :  Many  collaboration  reviews  and 
information  organization  tasks  were  not  included  in 
the  original  planning.  If  included,  the  project  took  1.8 
times  longer.  A  better  estimate  on  TRA  project 
timing  should  include  time  required  to  conduct 
reviews  and  information  gathering  and  organization. 

Lesson  # 2 :  The  time  required  to  identify  the 
technologies  for  each  contractor  can  be  significantly 
reduced  if  the  information  is  provided  by  the 
program  engineers  or  the  contractors. 

The  JLTV  has  17  physical  subsystems.  The  same 
Work  Breakdown  Structure  (WBS)  was  used  by  each 
contractor. 

Lesson  #3:  In  order  to  capture  all  potential 
technologies,  review  and  extract  all  technologies 
from  supporting  engineering  and  design  review 
documents  typically  categorized  by  vehicle  WBS. 
Specify  key  requirements  with  each  technology 
extracted. 


Lesson  #4;  Before  requesting  support  from  subject 
matter  experts  (SMEs)  and  the  PM,  develop  a 
flowchart  (Fig.  1)  to  show  the  TRA  process  and  a 
swimlane  chart  (Fig.  2  &  3)  to  show  the  detail  TRA 
steps  combined  with  roles  of  each  participating 
organizations.  These  charts  demonstrate  how  and 
when  team  members  have  to  rely  on  each  other  to 
complete  the  TRA  project. 


Figure  1  JLTV  TRA  Process 


There  were  5  organizations  involved  in  the  TRA 
process:  PM-JLTV,  TARDEC,  Army/USMC 

Independent  Review  Team  (IRT),  ASA(ALT)  IRT  and 
ASD(R&E)  and  contractor(s).  TARDEC  established 
the  IRT  by  selecting  SMEs  from  the  Army  and  USMC 
research  centers  since  JLTV  is  a  joint  program 
between  USA  and  USMC.  PM  selected  TARDEC  to 
provide  an  independent  assessment  of  the  CTEs 
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technical  risk/maturity.  ASA(ALT)  also  established  a 
separate  IRT  consisting  of  independent  experts  from 
the  academia,  industry  and  government  research 
centers. 


Figure  3  Technical  Risk  Assessment  Process 


CTE  Selection 

The  MS-B  Review  mainly  focused  on  the  CTEs  which 
are  defined  by  2  main  criteria.  First,  the  system 
being  acquired  depends  on  this  technology  element 
to  meet  Key  Performance  Parameters  (KPP)  or  Key 
System  Attributes  (KSA).  Second,  the  technology  or 
its  application  is  new  or  in  an  area  that  poses  major 
technological  risk  during  design/demonstration. 
These  criteria  are  expanded  upon  in  Table  2  to  assist 
the  CTE  identification  process. 


Table  2  CTE  Screening  Criteria 


Critical  Technology  Element  Screening  Criteria 

Y/ 

N 

Y/ 

N 

1)  Critical 

Directly  impact  an  operational  Requirement  threshold 
(KPP  or  WBS)? 

Significant  impact  on  an  improved  delivery  schedule? 

Significant  impact  on  program  affordability? 

AND 

2)  New  or  novel 
application 

Is  the  technology  new  or  novel? 

Is  the  technology  modified? 

Tech  repackaged  such  that  a  new  relevant  Environment  is 

realized? 

Expected  to  operate  beyond  design  intention  & 
demonstrated  capability? 

Or 

3)  High 

Risk 

Is  the  technology  in  an  area  that  poses  major 
technological  risk  during  detailed  design  or 
demonstration? 

Lesson  #5:  To  determine  all  key  requirements 
associated  with  a  technology,  set  up  a  functional 
requirement  tree  using  the  Capability  Development 


Document  (CDD)  with  identified  KPPs  and  KSAs 
and/or  Purchase  Description  (PD)  requirements  if 
available. 

The  PD  is  a  complete  set  of  system  requirements 
developed  from  the  CDD,  system  boundary  diagrams 
and  interface  analysis,  functional  analysis,  industry 
and  government  standards,  etc. 

Lesson  # 6 :  Each  system  requirement  should  be 
accompanied  by  acceptance  criteria  and  verification 
test  procedure.  Each  test  procedure  should  be  based 
on  some  functional  analysis  and  mission  profile.  If  it 
is  not,  the  program  can  be  at  risk  of  failing  some 
relevant  environment  verification  test  in  the  future. 

With  a  little  practice,  functional  and  non-functional 
requirements  can  be  easily  linked  to  any 
technologies. 

Lesson  #7:  In  order  to  separate  the  CTEs  from  non- 
critical  technologies,  an  independent  SME  has  to 
review  all  engineering  documents  such  as  test 
results,  failure  analysis  and  corrective  action  report 
(FACAR),  documented  design  risks  and  issues  matrix, 
requirement  compliance  matrix,  failure  mode  and 
effect  analysis  (FMEA),  modeling  and  simulation 
analysis,  interface  analysis,  trade  studies,  etc  to 
determine  if  there  was  any  major  technological  risk 
associated  with  a  technology. 

The  same  information  is  also  used  to  determine  if 
there  are  any  system  integration  risks  or  issues. 
Since  all  3  vendors  used  the  same  WBS,  their  design 
review  packages  were  similar.  Any  unique  or 
common  approaches  used  by  the  contractors,  issues 
and  difficulties  experienced  in  the  design  and 
integration  processes  and  disadvantages  or 
advantages  of  the  competing  technologies  can  be 
easily  compared.  After  ASA(ALT)  IRT  and  ASD(R&E) 
review  the  rationale  supporting  the  selection  of  the 
CTEs,  they  may  certify  all  or  some  of  the  CTEs 
suggested  by  the  PM/TARDEC  Team  and  provide 
additional  recommendations. 

Technical  Risk  Assessment 

TARDEC  selected  2  methods  to  assess  the  overall 
technical  risk  of  the  CTEs. 

The  first  assessment  was  done  primarily  by  the 
USA/USMC  IRT  based  on  a  detail  review  of 
contractor's  engineering  supporting  data  including 
test  results. 
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Lesson  #8:  Before  performing  the  assessment,  a 
detailed  metric  has  to  be  developed  and  agreed  upon 
between  the  program  and  USA/USMC  IRT.  The 
metric  should  contain  specific  detail  technical 
requirements  for  each  CTE,  verification  methods  and 
acceptance  criteria. 

It  is  very  tedious  to  locate  all  the  relevant 
engineering  reports,  test  results  and  procedure. 

Lesson  #9:  Rather  than  having  the  SMEs  to  locate  the 
engineering  information  they  need  to  make  the 
assessment,  locate  and  catalogue  all  relevant 
technical  data  (M&S  results,  test  data  and  reports, 
failure  incidences  and  corrective  actions,  engineering 
analysis,  etc.)  so  they  can  easily  refer  to  them  while 
performing  the  assessment. 

Additional  information  on  the  design,  integration 
and  latest  test  results  can  be  obtained  over  a  series 
of  meetings  between  the  program  and  the  SMEs. 
This  process  will  furnish  the  SMEs  with  all  necessary 
information  to  perform  the  technical  risk  assessment 
for  each  CTE. 

Since  the  beginning  of  the  test  program,  many 
system  issues  were  discovered.  A  system  can  fail 
because  of  design  issues,  manufacturing  flaw,  poor 
integration,  etc. 

Lesson  #10:  Failure  to  meet  requirement  does  not 
necessarily  mean  TRL  <  6.  In  case  of  non-compliance, 
the  SMEs  will  review  the  corrective  action  taken  and 
determine  if  there  is  an  unacceptable  risk  for  the 
EMD  Phase.  A  justification  will  be  required. 

Lesson  #11:  To  provide  a  more  comprehensive 
technical  risk  assessment  for  each  CTE,  in  addition  to 
Technology  Readiness  Level  (TRL,  Ref.  1),  it  is 
recommended  to  include  Manufacturing  Readiness 
Level  (MRL,  Ref.  2)  and  System  Readiness  Level  (SRL, 
Ref.  3)  in  the  assessment. 

The  definitions  of  TRL,  MRL  and  SRL  are  given  in  the 
Appendix.  Based  on  TRA  Guidance,  all  CTEs  entering 
EMD  Phase  shall  be  assessed  at  least  TRL6  (system 
prototype  demonstrated  satisfactorily  at  relevant 
environment).  It  is  not  a  DoD  requirement  that  the 
technology  is  assessed  at  MRL6.  However,  the 
sources  recommend  MRL  and  SRL  to  be  at  Level  6. 

Lesson  #12:  MS-B  Review  focus  is  on  technology 
maturity,  as  opposed  to  engineering  and  integration 
risk.  The  technical  risk  assessment  should  mainly 


focus  on  TRL  but  MRL  and  SRL  should  also  be 
mentioned  if  there  are  significant  manufacturing  and 
integration  issues  during  the  Technology 
Development  (TD)  Phase. 


An  example  of  a  boundary  diagram  is  given  in  Fig.  4. 
Figure  4  Typical  Boundary  Diagram 


Lesson  #13:  The  integration  of  a  technology  requires 
a  comprehensive  boundary  diagram  consisting  of  all 
the  external  and  internal  subsystems  that  have 
influence  on  the  technology  or  impacted  by  it.  Their 
relationships  shall  be  defined  in  detail  with  the 
understanding  that  these  interfaces  will  be 
demonstrated  in  a  lab  and  operational  environment. 

Most  technologies  have  the  following  types  of 
interfaces  which  require  specifications: 

Mechanical  interfaces  -  size,  weight,  clearance,  tool 
access,  etc. 

Electrical  interfaces  -  connectors,  cables,  energy 
storage,  power  consumption,  controls,  etc. 
Electromagnetic  interfaces  -  magnetic  fields,  radio 
and  optical  links,  etc. 

Thermal  interfaces  -  heat  production,  dissipation,  air 
conditioning  characteristics,  etc. 

Communication  interfaces  -  analog  and  digital 
signals  telecommunications,  transmission,  etc. 

Fluid  interfaces  -  air  conditioning,  compressed  air, 
exhaust  gas,  lubricating  oil,  fuel,  etc. 

Computer  interfaces  -  hardware/software,  protocol, 
users,  peripheral,  etc. 

Lesson  #14:  Many  of  the  RAM  &  performance  tests, 
failure  root  causes  and  corrective  action  reports  were 
not  complete  when  maturity  assessments  were 
made.  It  is  better  to  update  the  assessments  at  the 
end  of  the  test  phase  after  test  &  evaluation 
engineers  summarize  the  test  results. 

The  second  assessment  was  done  using  the  TRL 
calculator.  This  Excel  calculator  was  originally 
developed  by  the  Air  Force  Research  Lab  (AFRL).  For 
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each  maturity  level,  there  is  a  generic  list  of  tasks 
that  need  to  be  performed  to  bring  a  technology  to 
that  level  of  maturity.  By  specifying  the  percent 
completion  of  each  of  these  tasks,  the  calculator 
provides  a  repeatable  and  standardized  method  to 
determine  the  maturity  of  any  technology.  These 
tasks  are  divided  into  technical,  manufacturing  and 
programmatic  categories  for  each  level  of  maturity. 

Lesson  #15:  It  is  recommended  to  have  the  program 
engineers  to  complete  the  checklist  of  tasks  at  the 
end  of  the  test  phase  so  that  they  have  all  the 
information  needed. 

In  summary,  the  TRA  lessons  learned  from  the  18 
months  of  supporting  the  program  provide  a  detail 
roadmap  to  a  more  efficient  TRA  process  that  can 
significantly  reduce  the  time  to  perform  a  TRA. 
However,  this  does  not  mean  that  the  assessment 
can  start  at  a  much  later  time  because  it  may  take  a 
long  time  to  collect  the  required  substantiation  data 
to  assess  the  overall  technical  risks  associated  with  a 
technology. 


APPENDIX: 


TRL 

Technology  Readiness  Level  (TRA  Guidance,  ASD(R&E) 

1 

Basic  principles  observed  and  reported. 

2 

Technology  concept  and/or  application  formulated. 

3 

Analytical  and  experimental  critical  function  and/or 
characteristic  proof  of  concept. 

4 

Component  and/or  breadboard  validation  in  a  laboratory 
environment. 

5 

Component  and/or  breadboard  validation  in  a  relevant 
environment. 

6 

System/subsystem  model  or  prototype  demonstration  in 
a  relevant  environment. 

7 

System  prototype  demonstration  in  an  operational 
environment. 

8 

Actual  system  completed  and  qualified  through  test  and 
demonstration. 

9 

Actual  system  proven  through  successful  mission 
operations. 

MRL 

Manufacturing  Readiness  Level  for  Design  Thread  (MRL 
Deskbook) 

1 

Not  defined 

2 

Not  defined 

3 

Relevant  materials/processes  evaluated  for 

manufacturability  using  experiments/models. 

4 

Initial  producibility  and  manufacturability  assessment 
of  preferred  systems  concepts  completed.  Results 
considered  in  selection  of  preferred  design  concepts  and 
reflected  in  TDS  key  components/technologies. 

5 

Producibility  and  manufacturability  assessments  of  key 
technologies  and  components  initiated  as  appropriate. 
Ongoing  design  trades  consider  manufacturing  processes 
and  industrial  base  capability  constraints.  Manufacturing 
processes  assessed  for  capability  to  test  and  verify  in 
production,  and  influence  on  operations  &  support. 

6 

Producibility  assessments  and  producibility  trade  studies 

(performance  vs.  producibility)  of  key 

technologies/components  completed.  Results  used  to 
shape  Acquisition  Strategy,  SEP,  Manufacturing  and 
Producibility  plans,  and  planning  for  EMD  or  technology 
insertion  programs.  Preliminary  design 
choices  assessed  against  manufacturing  processes  and 
industrial  base  capability  constraints.  Producibility 
enhancement  efforts  (e.g.,  Design  for  Manufacturing 
Assembly,  Etc.  (DFX))  initiated. 

7 

Detailed  producibility  trade  studies  using  knowledge 
of  key  design  characteristics  and  related  manufacturing 
process  capability  completed.  Producibility 

enhancement  efforts  (e.g.,  DFX)  ongoing  for  optimized 
integrated  system.  Manufacturing  processes  reassessed 
as  needed  for  capability  to  test  and  verify  potential 
influence  on  operations  &  support. 

8 

Producibility  improvements  implemented  on  system. 
Known  producibility  issues  have  been  resolved  and  pose 
no  significant  risk  for  LRIP. 

9 

Prior  producibility  improvements  analyzed  for 
effectiveness  during  LRIP.  Producibility  issues/risks 
discovered  in  LRIP  have  been  mitigated  and  pose  no 
significant  risk  for  FRP. 

10 

Design  producibility  improvements  demonstrated 

In  FRP.  Process  producibility  improvements  ongoing.  All 
modifications,  upgrades,  DMSMS,  and  other  changes 
assessed  for  producibility. 

SRL 

System  Readiness  Level  (Australia  MoD) 

1 

Basic  principles  observed  and  reported. 

2 

System  concept  and/or  application  formulated. 

3 

Analytical  studies  and  experimentation  on  system 
elements. 

4 

Sub-system  components  integrated  in  a  laboratory 
environment. 

5 

System  tested  in  a  simulated  environment. 

6 

System  demonstrated  in  a  simulated  operational 
environment,  including  interaction  with  simulations  of 
external  systems 

7 

Demonstration  of  system  prototype  in  an  operational 
environment,  including  interaction  with  external 
systems. 

8 

System  proven  to  work  in  the  operational  environment, 
including  integration  with  external  systems. 

9 

Application  of  the  system  under  operational  mission 
conditions. 

TRL 

Level  of  Integration  Required  Corresponding  to  TRL 
(United  Kingdom  DoD) 

1 

Interface  requirements  and  impact  on  other  systems  are 
understood  at  concept  level  only. 

2 

Not  defined 

3 

Analytical  assessment  conducted  to  establish  interface 
requirements. 

4 

Interface  requirements  specified  and  understood.  The 
likely  impact  on  interfaced  systems  is  generally 
understood. 

5 

Interfaces  partially  demonstrated  at  System/Sub-system 
level  in  a  synthetic  environment.  Impact  on  other 
systems  is  understood,  specified  and  quantified. 

6 

Interfaces  demonstrated  at  system  level  in  a  synthetic  / 
high  fidelity  environment. 

7 

Fully  integrated  with  prototype.  System  interfaces 
qualified  in  an  operational  environment. 

8 

Final  production  design  validated  demonstrating  internal 
and  external  integration. 

9 

Not  defined 
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Unclassified:  Distribution  A.  Approved  for  public  release 
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